Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification

The amount of medical images for training deep classification models is typically very scarce, making these prone to overfit the data. Studies showed that knowledge distillation (KD), especially mean-teacher framework which more robust perturbations, can help mitigate over-fitting effect. However, directly transferring KD from computer vision image yields inferior performance as suffer higher intra-class variance and class imbalance. To address issues, we propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, takes commonly used model supervisor. Specifically, Class-guided (CCD) module pull closer positive pairs same in teacher student models, while pushing apart negative different classes. With this regularization, feature distribution shows similarity inter-class variance. Besides, Relation Preserving (CRP) loss distill teacher’s relational class-balanced manner. contribution CCD CRP, our CRCKD algorithm comprehensively. Extensive experiments on HAM10000 APTOS datasets demonstrate superiority proposed method. source code available at https://github.com/hathawayxxh/CRCKD.